Delivering a Single End User Experience to a Client from Multiple Servers

ABSTRACT

Methods, systems, and computer readable media are disclosed for creating a multi-port client server connection in a remote desktop environment. In an embodiment, the multi-client server connection can decouple client-server connection and client-server input from graphics delivery from the server to the client. Such embodiment provides higher server performance and enables dynamic server resource management. The multi-client server connection can be implemented in a virtual environment or between physical machines where one set of physical machines receives the graphics requests from the client and another set of physical machines generates and transmits display data to the client.

BACKGROUND

Remote computing systems may enable users to access resources hosted bythe remote computing systems. Servers on the remote computing systemscan execute programs and transmit signals indicative of a user interfaceto clients that can connect by sending signals over a network conformingto a communication protocol such as TCP/IP, UDP, or other protocols.Each connecting client may be provided a virtual desktop or a session,i.e., an execution environment that includes a set of resources. Eachclient can transmit signals indicative of user input to the server andthe server can apply the user input to the appropriate session. Theclients may use protocols such as the Remote Desktop Protocol (RDP) toconnect to a server resource.

With an increased availability of network bandwidth and an increaseddemand for rich 2D and 3D client graphics applications, there has been ashift in the remote computing system architectures. Instead of relyingpurely on local computing power, servers incorporate graphicsvirtualization platforms that shift graphics processing intelligence tohosted virtual desktop infrastructures (VDI) deployed in data centers.Clients experience virtual desktops in full fidelity, harnessing thegraphics processing power of shared graphics processing units (GPUs) andprocessors installed on the VDI. An example of a graphics virtualizationplatform is Microsoft® RemoteFX® that builds on a Hyper-V® VDIarchitecture and integrates RDP to deliver new payload designed forhosted VDI desktops.

A typical VDI architecture can include a host partition and a number ofguest partitions or virtual machines. The host partition has access tothe underlying physical resources of the VDI, such as GPUs, centralprocessing units (CPUs), and memory space, and can allocate and managethe access of the virtual machines to these resources. Each virtualmachine has a set of virtual resources that are a virtualization of theallocated physical resources.

As such, in a remote computing system, a client can connect to a virtualmachine or a virtual desktop session running therein, where anauthentication of the client is managed. Data such as user input data orgraphics data to be transmitted from the client to the virtual machineis initially transmitted to a network interface card (NIC) on the hostpartition and then re-routed to the virtual machine. The virtual machinecan use its virtual resources to process the data. The virtual machinesends the processed data to the host partition for further processing onthe underlying physical resources. The host partition further processesand sends the data back to the virtual machine for authentication withthe client. The virtual machine packages and re-routes the data back tothe host partition for transmission over the host partition NIC to theclient. The repetitive traversal of the data between the host partitionand the virtual machine can require intensive operations that canconsume significant amount of memory and CPU resources and that canincrease the data delivery latency to the client.

SUMMARY

Systems, methods, and computer readable media are disclosed foroptimizing the processing of data, such as graphics data, received fromclients in a remote computing system environment. Compared to currentarchitectures, such optimization includes a reduction in usage of memoryand CPU resources hosted and a reduction in data delivery latency to theclients.

In an embodiment, a client can initiate a first connection with anothercomputing device such as a compute server to execute the client workloadin a virtual desktop or a session therein. The compute server canauthenticate the first connection and can acquire the client address.The compute server can then negotiate a connection with a graphicsserver and can initialize and instruct the graphics server to expect aconnection originating from the client address. The compute server canalso acquire from the graphics server a graphics server address. Thecompute server can provide the graphics server address to the client. Inturn, the client can establish the third connection with the graphicsserver using the graphics server address. Once the connections areestablished, the client can provide a user's screen input, such askeyboard input, mouse input, and the like, to the compute server overthe first connection. The compute server can process the provided inputand output processed data such as display commands and calls. Thecompute server can send the processed data to the graphics server forprocessing. The graphics server can then process the data received andcan send graphics output data to the client. As such, there is no needfor the graphics server to send back the graphics output data to thecompute server. Similarly, there is no need for the compute server tosend the graphics output data to the client.

The compute server can be a guest partition or a virtual machine hostedon a virtualized computing system while the graphics server can be ahost partition on the virtualized computing system. The first connectioncan be, for example, a TCP/IP, a UDP, or any other network-basedcommunication and can comprise a remote desktop session connection. Thesecond connection can comprise, for example, an intra-partitioncommunication channel such as a VMBus, communication over a hypervisor(also known as a virtual machine monitor), a TCP/IP, a UDP, or any othernetwork-based connection. The third connection can be, for example, aTCP/IP, a UDP, FCOE, 100 GB Ethernet, or any other network-basedconnection.

In an embodiment, a plurality of clients can be simultaneously connectedto a plurality of compute servers and to a plurality of graphicsservers. At the initial step of establishing a connection with andauthenticating a client of the plurality of clients, a redirector and/ora broker can be used to connect the client to a first compute server ofthe plurality of compute servers. The redirector and/or broker candetermine availabilities of the plurality of compute servers andallocate the connection between the client and the first compute serveraccordingly. As such, in case of load balancing of compute resources,techniques such as virtual machine live migration can seamlesslytransfer the workload from the first compute server to a second computeserver. The client connection to the second compute server can bere-established while a connection between the client and a graphicsserver of the plurality of graphics servers can remain the same.Similarly, a graphics server manager can also be used to connect theclient connected compute server to a first graphics server of theplurality of graphics servers. The graphics server manager can determineavailabilities of the plurality of graphics servers and allocate theconnection between the client connected compute server and the firstgraphics server accordingly. As such, in case of load balancing ofgraphics servers, the client connected compute server can create a newconnection with a second graphics server and can request the client toestablish a new connection with the second graphics sever. The clientcan then seamlessly transition over to the second graphics server.

In an embodiment where a plurality of clients are simultaneouslyconnected to a plurality of compute servers and a plurality of graphicsservers, at least one client can be configured to receive rendered,captured, and compressed data from the plurality of graphics servers. Assuch, a user interfacing through the at least one client can view therendered, captured, and compressed data originating from the one or moreclients. Similarly, at least one graphics server can be configured totransmit rendered, captured, and compressed processed data originatingfrom the plurality of clients to one client.

This summary is intended to provide an overview of aspects of theinvention. It is not intended to identify any necessary steps orcomponents of the invention. In addition to the foregoing, other aspectsare described in the claims, drawings, and text forming a part of thepresent disclosure. It can be appreciated by one of skill in the artthat one or more various aspects of the disclosure may include but arenot limited to circuitry and/or programming for effecting theherein-referenced aspects of the present disclosure; the circuitryand/or programming can be virtually any combination of hardware,software, and/or firmware configured to effect the herein-referencedaspects depending upon the design choices of the system designer.

BRIEF DESCRIPTION OF THE DRAWINGS

The systems, methods, and computer media for optimizing the processingof data, such as graphics data, received in a remote computingenvironment in accordance with this specification are further describedwith reference to the accompanying drawings in which:

FIG. 1 depicts an example computing environment wherein aspects of thepresent disclosure can be implemented.

FIG. 2 depicts a remote computing environment for practicing aspects ofthe present disclosure.

FIG. 3 depicts a remote environment where a plurality of clients canconnect to a plurality of remote servers for practicing aspects of thepresent disclosure.

FIG. 4 depicts an example virtual machine environment, with a pluralityof virtual machines.

FIG. 5 depicts another example virtual machine environment, with aplurality of virtual machines.

FIG. 6 depicts a remote server hosting a plurality of virtual desktopsessions for practicing aspects of the present disclosure.

FIG. 7 depicts an example compute server and graphics serverarchitecture for practicing aspects of the present disclosure.

FIG. 8 depicts another example compute server and graphics serverarchitecture for practicing aspects of the present disclosure.

FIG. 9 depicts an example compute server and plurality of graphicsservers for practicing aspects of the present disclosure.

FIG. 10 depicts an example compute server and graphics servereffectuated in a virtual environment for practicing aspects of thepresent disclosure.

FIG. 11 depicts a computing environment with a client, compute server,and graphics server for practicing aspects of the present disclosure.

FIG. 12 depicts a computing environment with a plurality of clients,compute servers, and graphics servers for practicing aspects of thepresent disclosure.

FIG. 13 depicts a flow chart illustrating an example method forpracticing aspects of the present disclosure.

FIG. 14 depicts an example system and computer readable storage mediumfor practicing aspects of the present disclosure.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Certain specific details are set forth in the following description andfigures to provide a thorough understanding of various embodiments ofthe disclosure. Certain well-known details often associated withcomputing and software technology are not set forth in the followingdisclosure to avoid unnecessarily obscuring the various embodiments ofthe disclosure. Further, those of ordinary skill in the relevant artwill understand that they can practice other embodiments of thedisclosure without one or more of the details described below. Finally,while various methods are described with reference to steps andsequences in the following disclosure, the description as such is forproviding a clear implementation of embodiments of the disclosure, andthe steps and sequences of steps should not be taken as required topractice this disclosure.

It should be understood that the various techniques described herein maybe implemented in connection with hardware or software or, whereappropriate, with a combination of both. Thus, the methods and apparatusof the disclosure, or certain aspects or portions thereof, may take theform of program code (i.e., instructions) embodied in tangible media,such as floppy diskettes, CD-ROMs, hard drives, or any othermachine-readable storage medium wherein, when the program code is loadedinto and executed by a machine, such as a computer, the machine becomesan apparatus for practicing the disclosure. In the case of program codeexecution on programmable computers, the computing device generallyincludes a processor, a storage medium readable by the processor(including volatile and non-volatile memory and/or storage elements), atleast one input device, and at least one output device. One or moreprograms that may implement or utilize the processes described inconnection with the disclosure, e.g., through the use of an applicationprogramming interface (API), reusable controls, or the like. Suchprograms are preferably implemented in a high level procedural or objectoriented programming language to communicate with a computer system.However, the program(s) can be implemented in assembly or machinelanguage, if desired. In any case, the language may be a compiled orinterpreted language, and combined with hardware implementations.

The term circuitry used throughout the disclosure can include hardwarecomponents such as hardware interrupt controllers, hard drives, networkadaptors, graphics processors, hardware based video/audio codecs, andthe firmware/software used to operate such hardware. The term circuitrycan also include microprocessors configured to perform function(s) byfirmware or by switches set in a certain way or one or more logicalprocessors, e.g., one or more cores of a multi-core general processingunit. The logical processor(s) in this example can be configured bysoftware instructions embodying logic operable to perform function(s)that are loaded from memory, e.g., RAM, ROM, firmware, and/or virtualmemory. In example embodiments where circuitry includes a combination ofhardware and software an implementer may write source code embodyinglogic that is subsequently compiled into machine readable code that canbe executed by a logical processor. Since one skilled in the art canappreciate that the state of the art has evolved to a point where thereis little difference between hardware, software, or a combination ofhardware/software, the selection of hardware versus software toeffectuate functions is merely a design choice. Thus, since one of skillin the art can appreciate that a software process can be transformedinto an equivalent hardware structure, and a hardware structure canitself be transformed into an equivalent software process, the selectionof a hardware implementation versus a software implementation is left toan implementer.

Embodiments of the invention may execute on one or more computers. FIG.1 and the following discussion are intended to provide a brief generaldescription of a suitable computing environment in which the disclosuremay be implemented. One skilled in the art can appreciate that computersystems can have some or all of the components described herein below.

FIG. 1 depicts an example of a computing system which is configured towork with aspects of the disclosure. The computing system can include acomputer 100 or the like, including a logical processing unit 102, asystem memory 22, and a system bus 23 that couples various systemcomponents including the system memory to the logical processing unit102. The system bus 23 may be any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. The system memoryincludes read only memory (ROM) 24 and random access memory (RAM) 104. Abasic input/output system 26 (BIOS), containing the basic routines thathelp to transfer information between elements within the computer 100,such as during start up, is stored in ROM 24. The computer 100 mayfurther include a hard disk drive 27 for reading from and writing to ahard disk, not shown, a magnetic disk drive 28 for reading from orwriting to a removable magnetic disk 118, and an optical disk drive 30for reading from or writing to a removable optical disk 31 such as a CDROM or other optical media. In some example embodiments, computerexecutable instructions embodying aspects of the disclosure may bestored in ROM 24, hard disk (not shown), RAM 104, removable magneticdisk 118, optical disk 31, and/or a cache of logical processing unit102. The hard disk drive 27, magnetic disk drive 28, and optical diskdrive 30 are connected to the system bus 23 by a hard disk driveinterface 32, a magnetic disk drive interface 33, and an optical driveinterface 34, respectively. The drives and their associated computerreadable media provide nonvolatile storage of computer readableinstructions, data structures, program modules and other data for thecomputer 100. Although the environment described herein employs a harddisk, a removable magnetic disk 118 and a removable optical disk 31, itshould be appreciated by those skilled in the art that other types ofcomputer readable media which can store data that is accessible by acomputer, such as magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories (RAMs), read onlymemories (ROMs) and the like may also be used in the operatingenvironment.

A number of program modules may be stored on the hard disk, magneticdisk 118, optical disk 31, ROM 24 or RAM 104, including an operatingsystem 35, one or more application programs 36, other program modules 37and program data 38. A user may enter commands and information into thecomputer 100 through input devices such as a keyboard 40 and pointingdevice 42. Other input devices (not shown) may include a microphone,joystick, game pad, satellite disk, scanner or the like. These and otherinput devices are often connected to the logical processing unit 102through a serial port interface 46 that is coupled to the system bus,but may be connected by other interfaces, such as a parallel port, gameport or universal serial bus (USB). A display 47 or other type ofdisplay device can also be connected to the system bus 23 via aninterface, such as a GPU/video adapter 112. In addition to the display47, computers typically include other peripheral output devices (notshown), such as speakers and printers. The system of FIG. 1 alsoincludes a host adapter 55, Small Computer System Interface (SCSI) bus56, and an external storage device 62 connected to the SCSI bus 56.

The computer 100 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer49. The remote computer 49 may be another computer, a server, a router,a network PC, a peer device or other common network node, a virtualmachine, and typically can include many or all of the elements describedabove relative to the computer 100, although only a memory storagedevice 50 has been illustrated in FIG. 1. The logical connectionsdepicted in FIG. 1 can include a local area network (LAN) 51 and anetwork 52, which, as one example is a wide area network (WAN). Suchnetworking environments are commonplace in offices, enterprise widecomputer networks, intranets and the Internet.

When used in a LAN networking environment, the computer 100 can beconnected to the LAN 51 through a network interface controller (NIC) 114or adapter. When used in a WAN networking environment, the computer 100can typically include a modem 54 or other means for establishingcommunications over the network 52, such as the Internet. The modem 54,which may be internal or external, can be connected to the system bus 23via the serial port interface 46. In a networked environment, programmodules depicted relative to the computer 100, or portions thereof, maybe stored in the remote memory storage device. It will be appreciatedthat the network connections shown are examples and other means ofestablishing a communications link between the computers may be used.Moreover, while it is envisioned that numerous embodiments of thedisclosure are particularly well-suited for computer systems, nothing inthis document is intended to limit the disclosure to such embodiments.

In some instances, a user may desire to access computing applicationsremotely, i.e., applications that are running on a separate computingdevice. One implementation provides a user with such access through aremote desktop, such as a virtual desktop. Embodiments of a remotedesktop system may execute one or more computers or may have some or allof the components described with respect to computer 100 of FIG. 1. Aremote desktop system is a computer system that maintains applicationsthat can be remotely executed by and displayed on client computersystems. FIG. 2 depicts an example architecture of a remote desktopsystem 200. The remote desktop system 200 can comprise a remote clientcomputer 210 and a remote server computer 220. The remote clientcomputer 210 and the remote server computer 220 are configured toconduct a remote session such as a virtual desktop session with eachother.

As depicted, the remote server computer 220 serves a remote session tothe remote client computer 210 where the remote server computer 220sends client graphical output from executing user a remote clientsession 222. A remote user input is entered at the remote clientcomputer 210. An input manager 212 can process and transfer the remoteuser input over a network (e.g., using protocols based on theInternational Telecommunications Union (ITU) T.120 family of protocolssuch as Remote Desktop Protocol (RDP)) to a remote user application 224on the remote server computer 220. The network may be any type ofcommunications network, such as a local area network, wide area network,cable network, the internet, the World Wide Web or a corporateenterprise network. The remote user application 224 can be executed in aremote client session 222 hosted on the remote server computer 220. Theremote user application 224 processes the input as if the input wereentered at the remote server computer 220. The remote user application224 generates remote server output in response to the received input andthe output is transferred over the network to the remote client computer210. The remote client computer 210 presents the output data to a remoteuser. Thus, input is received and output is presented at the remoteclient computer 210, while processing actually occurs at the remoteserver computer 220.

In addition to the remote user application 224, the remote clientsession 222 can include a shell and a user interface such as a desktop,the subsystems that track mouse movement within the desktop, thesubsystems that translate a mouse click on an icon into commands thateffectuate an instance of a program, other applications, etc. It shouldbe understood that the foregoing discussion is exemplary and that thepresently disclosed subject matter may be implemented in variousclient/server environments and not limited to a particular remotepresentation product.

In most, if not all remote desktop environments, the remote user inputdata (entered at the remote client computer 210) typically includesmouse and keyboard data representing commands to an application. Outputdata (generated by the remote user application at the remote servercomputer 220) typically includes graphics data for display at the remoteclient computer 210. Many remote desktop environments also includefunctionality that extends to transfer other types of data. In anexample embodiment, graphics data output from the user application 224can be sent to a graphics manager 226 hosted on the remote servercomputer 220. The graphics manager 226 can render, capture, compress,and transfer the graphics data over the network to a remote user display214 on the remote client computer 210. The remote user display 214 candisplay the graphics output data to a remote user.

In an embodiment of a remote desktop environment, a remote servercomputer may execute a plurality of remote sessions (or virtualdesktops) for a plurality of remote client computers. As such, a brokermay be used to control the allocation of sessions to the plurality ofremote client computers. Additionally, in a remote desktop environment,there may be a plurality of remote server computers that can serve aparticular remote client computer. As such, a redirector may be used tocontrol the allocation of remote server computers serving the particularremote client computer. FIG. 3 depicts an example embodiment of suchremote desktop system 300.

The plurality of remote client computers 310(A-N) may be any computingdevices capable of communicating with a remote server system 350 over anetwork, such as the remote client computer 210 of FIG. 2. The remoteserver system 350 may comprise a redirector 330, a broker 340, and aplurality of remote server computers 320(A-N). The redirector 330 andthe broker 340 may be computing devices that include processors andmemories configured to implement the respective functionalities of thesedevices as described herein below. The remote server computers 320(A-N)may have some or all of the components described with respect tocomputer 100 of FIG. 1 and remote server computer 220 of FIG. 2. Theremote server computers 320(A-N) may also be implemented as virtualmachines. The virtual machines can be executed on a single hardwareinfrastructure or on separate hardware infrastructures. The broker 340may be a standalone device connected to the redirector 330 using agateway (not shown), may be disposed within the redirector 330, or maybe disposed within the remote server computers 320(A-N). The redirector330 may also be disposed in the remote server computers 320(A-N).

The broker 340 allocates a session to a remote client computer based onsession state information stored in the broker 340. Session stateinformation may include, for example, session IDs, user names, names ofthe remote server computers where sessions are residing, the number ofactive sessions in each remote server computer, and so on. As usedherein a session may be a virtual desktop session (also known as virtualmachine session).

A remote client computer 310 first connects to the redirector 330 thatmay provide load balancing of remote client computers 310(A-N). In sucha case, the redirector 330 typically first receives the request for aconnection. The redirector 330 then accepts the connection request andqueries the broker 340 to determine where the remote client computer 310be redirected. The broker 340 analyzes the session state information ofthat particular environment and identifies a remote server computer 320to which the remote client computer 310 can be redirected. Theidentified remote server computer 320 may possess a session previouslyaccessed by the remote client computer 310, but later disconnected, towhich the remote client computer 310 can be reconnected again. In anembodiment, an identified remote server computer 320 may provide a newsession to which the remote client computer 310 can be connected,provided the remote client computer 310 does not possess any otherexisting sessions.

The broker 340 sends information to the requested remote server computer320 enabling the remote client computer 320 to establish a connectionwith the identified remote server computer 310. For example, theinformation may include a machine ID, a session ID, and location of theidentified remote server computer 320. Once the remote client computer310 establishes the connection with the identified remote servercomputer 320, the remote client computer 310 can access applicationspresent in the identified remote server computer 320. These applicationsmay be compatible to the logic of the broker 340 that was used inidentifying the remote server computer 320 in the remote server system350.

In an embodiment, the systems described above may be used to connect,for example, a remote client computer 310 to one of a plurality ofvirtual desktops or sessions therein running on a remote servercomputer. The remote client computer examines a redirector token in aremote desktop protocol (RDP) packet. The remote client computerconnects to one of the many virtual desktops based on informationcontained in the redirector token.

In another embodiment, a remote client computer 310 can be connected toone of the virtual desktops using the broker 340 and a pool manager (notshown). The pool manager may be disposed within the broker 340. Thebroker 340 assigns the virtual desktops to the remote client computer310 when the remote client computer 310 is connected to a virtualdesktop hosted on a virtual machine (VM), and the pool manager indicateswhich of the virtual desktops are available to be assigned.

In a further embodiment, the remote client computer 310 can be connectedto a virtual desktop. The remote client computer 310 indicates a networkname that is used by the broker 340 to generate an internet protocol(IP) address and to establish connection between the remote clientcomputer 310 and the virtual desktops. By hiding the individual virtualdesktop IP addresses from the remote client computers 310(A-N), only asingle network name of the broker 340 is initially needed to beexternally exposed to the remote client computers 310(A-N).

FIG. 4 illustrates an example virtual machine environment, with aplurality of virtual machines. A virtualized computer system 400 can beused to implement the remote server computer 220 of FIG. 2 and theremote server computers 320(A-N) of FIG. 3.

As shown in FIG. 4, computer system 400 can include elements describedin FIG. 1 and components operable to effectuate virtual machines. Onesuch component is a hypervisor microkernel 402 that may also be referredto in the art as a virtual machine monitor. The hypervisor microkernel402 can be configured to control and arbitrate access to the hardware ofcomputer system 400. The hypervisor microkernel 402 can generateexecution environments called partitions such as guest partition 1through guest partition N (where N is an integer greater than 1). Here,a guest partition is the basic unit of isolation supported by hypervisormicrokernel 402. A guest partition may also be known as a childpartition. Hypervisor microkernel 402 can isolate processes in onepartition from accessing another partition's resources. Each guestpartition can be mapped to a set of hardware resources, e.g., memory,devices, processor cycles, etc., that is under control of the hypervisormicrokernel 402. In embodiments hypervisor microkernel 402 can be astand-alone software product, a part of an operating system, embeddedwithin firmware of the motherboard, specialized integrated circuits, ora combination thereof.

Hypervisor microkernel 402 can enforce partitioning by restricting aguest operating system's view of the memory in a physical computersystem. When hypervisor microkernel 402 instantiates a virtual machine,it can allocate pages, e.g., fixed length blocks of memory with startingand ending addresses, of system physical memory (SPM) to the virtualmachine as guest physical memory (GPM). Here, the guest's restrictedview of system memory is controlled by hypervisor microkernel 402. Theterm guest physical memory is a shorthand way of describing a page ofmemory from the viewpoint of a virtual machine and the term systemphysical memory is shorthand way of describing a page of memory from theviewpoint of the physical system. Thus, a page of memory allocated to avirtual machine will have a guest physical address (the address used bythe virtual machine) and a system physical address (the actual addressof the page).

A guest operating system may virtualize guest physical memory. Virtualmemory is a management technique that allows an operating system to overcommit memory and to give an application sole access to a contiguousworking memory. In a virtualized environment, a guest operating systemcan use one or more page tables to translate virtual addresses, known asvirtual guest addresses into guest physical addresses. In this example,a memory address may have a guest virtual address, a guest physicaladdress, and a system physical address.

In the depicted example, the computer system 400 includes a hostpartition that can also be thought of as similar to domain 0 of Xen'sopen source hypervisor. The host partition may also be referred as aparent partition or a root partition. As depicted, the host partitioncan include a host 404. The host 404 can include device drivers 424 thatallow the interaction of applications with the underlying physicalhardware of computer system 400. As such, the host 404 can have accessto physical hardware of computer system 400, such as logical processingunit 102, GPU 112, and NIC 114. The host 404 can be an operating system(or a set of configuration utilities).

The host 404 can be configured to provide resources to guest operatingsystems executing in the guest partitions 1-N by using virtualizationservice providers 428 (VSPs). VSPs 428, which are typically referred toas back-end drivers in the open source community, can be used tomultiplex the interfaces to the hardware resources by way ofvirtualization service clients (VSCs) (typically referred to asfront-end drivers in the open source community or paravirtualizeddevices). As shown in FIG. 4, virtualization service clients executewithin the context of guest operating systems. However, these driversare different than the rest of the drivers in the guest in that they maybe supplied with a hypervisor, not with a guest. In an exemplaryembodiment the path used to by virtualization service providers 428 tocommunicate with VSCs 416 and 418 can be thought of as thevirtualization path.

The host partition VSPs 428 and the guest partition VSCs 416 and 418 canconnect to a virtual machine bus (VMBus). The VMBus is a logical channelwhich enables inter-partition communication. The guest partitionsrequests to the virtual resources, such as the virtual processors 430and 432, can be redirected via the VMBus to the devices in the hostpartition which can manage the requests. The response from the hostpartition to the guest partition can also be redirected via the VMBus.This entire process can be transparent to the guest OSes 220 and 222. Inanother embodiment, the host partition VSPs 228 and the guest partitionVSCs 216 and 218 can communicate over a network, such as a TCP/IPnetwork, by sending and receiving message packets.

As shown in FIG. 4, emulators 434, e.g., virtualized IDE devices,virtualized video adaptors, virtualized NICs, etc., can be configured torun within the host 404 and are attached to resources available to guestoperating systems 420 and 422. For example, when a guest OS touches amemory location mapped to where a register of a device would be ormemory mapped device, hypervisor microkernel 402 can intercept therequest and pass the values the guest attempted to write to anassociated emulator. Here, the resources in this example can be thoughtof as where a virtual device is located. The use of emulators in thisway can be considered the emulation path. The emulation path isinefficient compared to the virtualized path because it requires moreCPU resources to emulate device than it does to pass messages betweenVSPs and VSCs. For example, the hundreds of actions on memory mapped toregisters needed in order to write a value to disk via the emulationpath may be reduced to a single message passed from a VSC to a VSP inthe virtualization path.

Each guest partition can include one or more virtual processors (430 and432) that guest operating systems (420 and 422) can manage and schedulethreads to execute thereon. Generally, the virtual processors areexecutable instructions and associated state information that provides arepresentation of a physical processor with a specific architecture. Forexample, one virtual machine may have a virtual processor havingcharacteristics of an Intel x86 processor, whereas another virtualprocessor may have the characteristics of a PowerPC processor. Thevirtual processors in this example can be mapped to processors of thecomputer system such that the instructions that effectuate the virtualprocessors will be backed by processors. Thus, in an embodimentincluding multiple processors, virtual processors can be simultaneouslyexecuted by processors while, for example, other processor executehypervisor instructions. The combination of virtual processors andmemory in a partition can be considered a virtual machine.

Guest operating systems (420 and 422) can be any operating system suchas, for example, operating systems from Microsoft®, Apple®, the opensource community, etc. The guest operating systems can includeuser/kernel modes of operation and can have kernels that can includeschedulers, memory managers, etc. Generally speaking, kernel mode caninclude an execution mode in a processor that grants access to at leastprivileged processor instructions. Each guest operating system can haveassociated file systems that can have applications stored thereon suchas remote services or virtual desktop sessions, terminal servers,e-commerce servers, email servers, etc., and the guest operating systemsthemselves. The guest operating systems can schedule threads to executeon the virtual processors and instances of such applications can beeffectuated.

FIG. 5 depicts similar components to those of FIG. 4. However, in thisexample embodiment hypervisor 542 can include a microkernel componentand components similar to those in host 404 of FIG. 4 such as thevirtualization service providers 428 and device drivers 424, whilemanagement operating system 540 may contain, for example, configurationutilities used to configure hypervisor 542. In this architecture,hypervisor 542 can perform the same or similar functions as hypervisormicrokernel 402 of FIG. 4 and host 404. Hypervisor 542 can be astandalone software product, a part of an operating system, embeddedwithin firmware of a motherboard, and/or a portion of hypervisor 542 canbe effectuated by specialized integrated circuits.

In various embodiments, a remote server computer, such as the remoteserver computer 220 of FIG. 2, can execute multiple remote clientsessions or virtual desktops. Each remote client session, such as theremote client session 222, can represent an application environment fora connecting client. A remote server computer can generate at least oneremote client session for each of the connecting remote client computersas illustrated in FIG. 6. Moreover, as noted above, the remote servercomputer 220 may be a virtual machine executing some or all of thecomponents of computer system 400 of FIGS. 4 and 5 that, in turn,executes multiple remote client sessions.

Depicted in FIG. 6 is computer system 600, which may include circuitryconfigured to effectuate a remote server computer, or in otherembodiments the computer system 600 can include circuitry configured tosupport remote desktop connections. In the depicted example, thecomputer system 600 can be configured to generate one or more remoteclient sessions for connecting clients such as sessions 1 through N(where N is an integer greater than 2). Briefly, a session in exampleembodiments of the present invention can generally include anoperational environment that is effectuated by a plurality ofsubsystems, e.g., software code, that are configured to interact with akernel 614 of computer system 600. For example, a session can include aprocess that instantiates a user interface such as a desktop window, thesubsystems that track mouse movement within the window, the subsystemsthat translate a mouse click on an icon into commands that effectuate aninstance of a program, etc. A session can be generated by the computersystem 600 on a user by user basis by the computer system 600 when, forexample, the computer system 600 receives a connection request over anetwork connection from a client, such as the remote client computer 210of FIG. 2. Generally, a connection request can first be handled by thetransport logic 610 that can, for example, be effectuated by circuitryof the computer system 600. The transport logic 610 can in someembodiments include a network adaptor; firmware, and software that canbe configured to receive connection messages and forward them to theengine 612. As illustrated by FIG. 6, the transport logic 610 can insome embodiments include protocol stack instances for each session.Generally, each protocol stack instance can be configured to route userinterface output to a client and route user input received from theclient to the session core 644 associated with its session.

Continuing with the general description of FIG. 6, the engine 612 insome example embodiments of the present invention can be configured toprocess requests for sessions; determine the functionality for eachsession; generate sessions by allocating a set of physical resources forthe session; and instantiating a protocol stack instance for thesession. In some embodiments the engine 612 can be effectuated byspecialized circuitry components that can implement some of the abovementioned operational procedures. For example, the circuitry in someexample embodiments can include memory and a processor that isconfigured to execute code that effectuates the engine 612. As depictedby FIG. 6, in some instances the engine 612 can receive connectionrequests and determine that, for example, a license is available and asession can be generated for the request. In the situation where thecomputer system 600 is a remote computer that includes remote desktopcapabilities, the engine 612 can be configured to generate a session inresponse to a connection request without checking for a license. Asillustrated by FIG. 6, a session manager 616 can be configured toreceive a message from an engine 612 and in response to the message thesession manager 616 can add a session identifier to a table; assignmemory to the session identifier; and generate system environmentvariables and instances of subsystem processes in memory assigned to thesession identifier.

As illustrated by FIG. 6, the session manager 616 can instantiateenvironment subsystems such as a runtime subsystem 640 that can includea kernel mode part such as the session core 644. For example, theenvironment subsystems in an embodiment are configured to expose somesubset of services to application programs and provide an access pointto the kernel of the computer operating system 602. In exampleembodiments the runtime subsystem 640 can control the execution ofprocesses and threads and the session core 644 can send requests to theexecutive of the kernel 614 to allocate memory for the threads andschedule time for them to be executed. In an embodiment the session core644 can include a graphics display interface 646 (GDI), a securitysubsystem 650, and an input subsystem 652. The input subsystem 652 canin these embodiments be configured to receive user input from a clientvia the protocol stack instance associated with the session and transmitthe input to the session core 644 for the appropriate session. The userinput can in some embodiments include signals indicative of absoluteand/or relative mouse movement commands, mouse coordinates, mouseclicks, keyboard signals, joystick movement signals, etc. User input,for example, a mouse double-click on an icon, can be received by thesession core 644 and the input subsystem 652 can be configured todetermine that an icon is located at the coordinates associated with thedouble-click. The input subsystem 652 can then be configured to send anotification to the runtime subsystem 640 that can execute a process forthe application associated with the icon.

In addition to receiving input from a client, draw commands can bereceived from applications and/or a desktop and be processed by the GDI646. The GDI 646 in general can include a process that can generategraphical object draw commands. The GDI 646 in this example embodimentcan be configured to pass The GDI 646 output to the remote displaysubsystem 654 where the commands are formatted for the display driverthat is attached to the session. In certain example embodiments, one ormore physical displays can be attached to the computer system 600, e.g.,in a remote desktop situation. In these example embodiments, the remotedisplay subsystem 654 can be configured to mirror the draw commands thatare rendered by the display driver(s) of the remote computer system andtransmit the mirrored information to the client via a stack instanceassociated with the session. In another example embodiment, the remotedisplay subsystem 654 can be configured to include virtual displaydriver(s) that may not be associated with displays physically attachedto the computer system 600, e.g., the computer system 600 could berunning headless. The remote display subsystem 654 in this embodimentcan be configured to receive draw commands for one or more virtualdisplays and transmit them to the client via a stack instance associatedwith the session. In an embodiment of the present invention, the remotedisplay subsystem 654 can be configured to determine the displayresolution for each display driver, e.g., determine the displayresolution of the virtual display driver(s) associated with virtualdisplays or the display resolution of the display drivers associatedwith physical displays; and route the packets to the client via theassociated protocol stack instance.

In some example embodiments, the session manager 616 can additionallyinstantiate an instance of a logon process associated with the sessionidentifier of the session that can be configured to handle logon andlogoff for the session. In these example embodiments drawing commandsindicative of the graphical user interface associated with the logonprocess can be transmitted to the client where a user of the client caninput an account identifier, e.g., a username/password combination, asmart card identifier, and/or biometric information into a logon screen.The information can be transmitted to computer system 600 and routed tothe engine 612 and the security subsystem 650 of the session core 644.For example, in certain example embodiments the engine 612 can beconfigured to determine whether the user account is associated with alicense; and the security subsystem 650 can be configured to generate asecurity token for the session.

As described by FIG. 6, a remote server computer can provide multipleremote desktop sessions to connecting remote client computers. Theremote desktop sessions may be associated with one or more applicationsrequested by a remote client computer. Additionally and as described byFIG. 6, the remote server computer can process the graphics datarepresentative of a client desktop, such as a user interface screen, auser input commands, etc. Further, the remote server computer mayrender, capture, compress, and transmit the graphics data to the clientremote computer. Rendering refers to the process of translating rawdisplay calls, such as rotate, flip, and draw, made by the applicationsrunning within the remote desktop sessions. Capturing refers to theprocess of taking a rendered application content, such as on-screenbitmaps or frame changes, and intelligently capturing the change over aprevious rendering of the application content. Compressing, alsoreferred to as encoding, refers to the process of optimally andequitably delivering graphics resources to the each of the connectedremote client computer. The quality of network conditions and targetremote client computer determine the type of compression/encoding usedto optimally deliver captured content.

As will be described herein below, in various embodiments, the remoteserver computer may comprise a compute server and a graphics server. Thecompute serve can be configured to receive graphics data from a remoteclient computer, process the graphics data, and send the processedgraphics data to the graphics server. The graphics server can beconfigured to render, capture, and compress the received data from thecompute server into graphics output data. Rather than sending thegraphics output data to the compute server and using the compute serverto transmit the graphics output data to the remote client computer, thegraphics server may also be configured to transmit directly the graphicsoutput data to the remote client computer.

FIG. 7 depicts an example embodiment of a remote server computer 700comprising a compute server 710 and graphics server 720. Embodiments ofthe remote server computer 700 may execute some or all of the componentsdescribed with respect to computer 100 of FIG. 1, remote server computer220 of FIG. 2, remote server computer 320 of FIG. 3, computer system 400of FIGS. 4 and 5, and computer system 600 of FIG. 6.

Embodiments of the compute server 710 may execute some or all of thecomponents described with respect to computer 100 of FIG. 1, remoteserver computer 220 of FIG. 2, computer system 400 or virtual machine440 of FIGS. 4 and 5, and computer system 600 of FIG. 6. The computeserver 710 may also have insufficient or no GPU resources. In a furtherembodiment, the compute server 710 may be a standard server computerconfigured appropriately for providing the computing resources describedbelow. In another embodiment, the compute server 710 may be a computingdevice configured for specific functions. For example, a compute servermay have a single type of processing unit and a small amount of cachememory only.

The graphics server 720 may be configured to provide resources forgraphics operations, such as rendering, capturing, and compressingoperations. The graphics server may also be configured with a pluralityof GPU resources. In an embodiment, the graphics server 720 may executesome or all of the components described with respect to computer 100 ofFIG. 1. In a further embodiment, the graphics server may be hosted on ahost partition, such as host 404 of FIG. 4. The compute server 710 andthe graphics server 720 may be connected via a network (fiber channel,LAN, wireless, Ethernet, etc.). In a virtualized environment such as theenvironment of FIG. 4, the graphics server 720 and the compute server710 may also be connected using a VMBus.

The compute server 710 may run one or more applications 712. In oneaspect the application may be associated with a graphics device driver714. The graphics device driver 714, the application 712, and/or thecompute server 710 may be associated with a graphics server manager 740on the graphics server 720. The graphics device driver 714, theapplication 712, and/or the compute server 710 may be able to send andreceive instructions and data to and from the graphics server manager740. As one example, the graphics device driver 714, the application712, and/or the compute server 710 may be able to send first data to thegraphics server manager 740, the first data indicative of a request forGPU resources. The graphics server manager 740 may send second data tothe graphics device driver 714, the application 712, and/or the computeserver 710, the second data indicating routing for GPU instructions fromthe graphics server 720.

The graphics server manager 740 may manage the graphics server 720. Thegraphics server manager 740 may be able to send instructions and data tocomponents of the graphics server 720 and may receive information anddata as a response from components of the graphics server 720. Thegraphics server 720 may be specialized for GPU hosting and processing.The graphic server 720 may comprise the graphics server manager 740, aproxy graphics application 722, a kernel 726, and GPU hardware 730. Theproxy graphics application 722 may be associated with a first graphicsdevice driver 724, and the kernel 726 may be associated with a secondgraphics device driver 728. The graphics device driver 724 and 728 maytranslate, receive, and send data and information associated withgraphics processing tasks. In one embodiment, the graphics device driver724 and 728 are selected to translate between particular GPU hardware730 and the applications, hardware, and operating systems on thegraphics server manager 740, the compute server 710, and/or a remoteclient computer.

In the embodiment of FIG. 7, instructions associated with graphicsprocessing tasks can flow through a series of layers, from theapplication 712 to the graphics server manager 740, to the proxygraphics application 722, to the Kernel 726, to the hardware 730. Theprocessed information may follow the same path in reverse. FIG. 8illustrates an alternative embodiment for the information path. Thegraphic server manager 850 receives a request for GPU resources from thecompute server 710, the application 712, and/or the graphics devicedriver 714, and sends routing instructions, state instructions, and thelike to the compute server 710 and the graphics server 720. Thereafter,GPU tasks, processed information, and instructions may be sent directlybetween the graphics server 720 and the compute server 710. The graphicsserver manager 850 may monitor the interactions and may perform othertasks related to the allocation of resources on a GPU such as GPUhardware 730.

FIG. 9 depicts a plurality of graphics servers 720(A-N), which may beused when resources associated with a single set of GPU hardware 730 areinsufficient to perform a GPU processing task. This embodiment may alsobe used when a graphics server manager 850 migrates a part of a GPUprocessing task from a first graphics server 720A to a second graphicsserver 720B. In such an embodiment, the graphics server manager 850 mayact to copy the state of the first graphics server 720A to the secondgraphics server 720B.

FIG. 10 depicts an example embodiment of a remote server computer 1000with a compute server 1010 and a graphics server 1020 implemented in avirtualized environment or a virtual desktop infrastructure (VDI). Thecompute server 1010 and the graphics server 1020 can be configured toeffectuate the compute server 710 and graphics server 720 of FIGS. 7-9.Embodiments of the remote server computer 1000 may execute some or allof the components described with respect to computer 100 of FIG. 1,remote server computer 220 of FIG. 2, remote server system 350 of FIG.3, computer system 400 of FIGS. 4 and 5, and computer system 600 of FIG.6.

FIG. 10 depicts a host-based graphics virtualization for VDI. Sucharchitecture can, for example, be implemented using the Microsoft®RemoteFX® platform. Virtual machines 1011(A-N) can be referred to asvirtual desktops. The architectures uses virtual graphics processingunit (vGPU) 1016(A-N), which abstracts the relationship between guestoperating system (OS) 1014(A-N) and physical GPU 112 to optimally shareGPU resources in a hosted multi-user environment.

The hypervisor microkernel 1002 can be configured to effectuate aplurality of host and guest partitions. In a further embodiment, thehypervisor microkernel 1002 can integrate remote desktop sessioncomponents (not shown).

Each host partition can be configured as a graphics server 1020 that hasaccess to physical GPU resources of the remote computer server 1000.Each host partition can also include management components for graphicsrendering, capturing, and encoding. Each host partition can also includedevice drivers 1026 that provide an interface to physical GPUs 112 andto host-based encoders, such as ASICS (not shown). The device drivers1026 can include GPU, CPU, and encoder specific drivers.

Each guest partition can be configured as a compute server 1010 that hasaccess to vGPU resources. Each guest partition can effectuate one ormore virtual desktops or sessions for a plurality of connected remoteclient computers (not shown). The connection between each guestpartition and the remote client computers can comprise a remote desktopsession, such as RDP 7.1, similar to what is shown in FIGS. 2-3.

The vGPU 1016 can provide a virtual graphics adapter installed in eachvirtual machine 1011. The vGPU 1016 can abstract graphic processing formultiple virtual machines utilizing one or more GPUs 112. When anapplication running in the virtual machine 1011 invokes a graphicsoperation, such as a DirectX® or a GDI operation, the vGPU 1016 can usea communications channel between the guest partition 1010 and the hostpartition 1020 to obtain resources from the GPUs 112. The communicationchannel can comprise a VMBus or a TCP/IP channel. The VMBus can beconfigured within the hypervisor microkernel 1002 for memory sharing andother functions specific to the vGPU. Such VMBus configuration canprovide an integration mechanism directly into the hypervisormicrokernel 1002, where all resource requests for graphics-relateddevices can be transferred.

The vGPU 1016 can also provide a quality of service mechanism to virtualmachines 1011. The quality of service mechanism can equitably deliverGPU 112 resources to the virtual machines 1011 based on load-balancingpolicies that make most efficient use of the GPU 112.

The host partition 1020 can be configured to provide remote desktopvirtual graphics management (RDVGM) functions to the virtual machines1011. The RDVGM can manage resource assignment and process controlbetween physical resources of the remote computer server 1000 and vGPU1016 resource assignment into each virtual machine guest operatingsystem (OS) 1014. The RDVGM functions can include: managing therendering, capturing, and compressing (RCC) processes, assigning GPU 112resources to virtual machines 1011 through the vGPU 1016, assigningresource policies to virtual machines 1011, and load-balancing GPU 112resources across multiple virtual machines 1011(A-N). The RDVGM can alsoassign appropriate GPU 112 resources to virtual machines 1011(A-N) atboot time.

The RDVGM can integrate an RCC engine 1022, which handles rendering,capturing, and compressing of graphics data. The RCC can receivegraphics requests as output from each virtual machine 1011, andtranslate these requests into, for example, DirectX® compliant commandson the host partition 1020. A VMBus can provide a high-speedcommunications backplane for graphics requests from hosted applications1012(A-N) running in the virtual machines 1011(A-N) to physical GPU 112resources. For DirectX® compliant commands, applications 1012(A-N) needto support DirectX® 9 or later while GPUs 112 need to support DirectX®10 or later.

As previously described, rendering refers to the process of translatingraw display calls such as rotate, flip, and draw, made by theapplications 1012(A-N) through the vGPUs 1016(A-N), honoring thoserequests to the GPUs 112, and thus rendering application content. Therendering can be based on standard DirectX® syntax. Capturing refers tothe process of taking rendered application content, such as on-screenbitmap or frame changes, and intelligently capturing the change over aprevious rendering of the application content. A secondary function ofcapturing is assigning quality of service policies for capture qualityand encoding level. Compressing or encoding refers to the process ofoptimally and equitably delivering GPU 112 resources through the vGPU1016 to remote client computers over communication channels comprising,for example, a remote desktop session protocol. The quality andcondition of the communication channel and the type of the targetedremote client computer can determine the type of compression/encodingused to optimally deliver the captured content.

When an application 1012 running within a VM 1011 issues display callssuch as draw, resize, and rotate, the vGPU 1016 can broker all renderingrequests. The virtualization path can be transparent to the guest OS1014. As previously explained, the graphics processing commands can beintercepted by the host partition 1020. The interception can be done ata low level in the software stack. The graphics can then be rendered onthe GPUs 112 into a single frame buffer, which serves as a temporaryholding station for graphical updates. The frame buffer can representthe virtualized display of an end-user; the end-user being a connecteduser to the virtual machine 1011 using a remote client computer. Richgraphics applications, 3D plug-ins, and other graphics calls andcommands can run exactly as if the applications were running on adedicated workstation containing GPUs.

The host partition 1020 can rapidly and efficiently capture the renderedcontent. Each frame within the content can be divided into manageableunits. Change regions within the frame can be processed through anoptimization capability provided by the RCC engine 1022. Through thiscapturing mechanism, individual frames are intercepted for displaychanges. Only the regions within a frame that have changed are capturedfor encoding.

The host partition 1020 can also compress the captured content. Thecompression process can be controlled through administrative tools,giving certain virtual machines higher or lower priority, or dynamicallyby the size of the change regions within a captured frame. In addition,in an embodiment that uses a remote desktop session, such as RDP 7.1,the remote desktop session can provide frame-rate optimization based onnetwork usage and fairness. The RCC engine 1022 can reach out to aremote desktop session listener process to assess the state of a remoteclient computer, including the remote client computer decodingcapabilities. Changes to the frame buffer can be sent to the remoteclient computer at a frame rate that dynamically adapts to networkconditions and the remote client computer's ability to consume thechanges. The encoded output can be tunneled within the remote desktopsession and can be sent out to the remote client computer.

All the architecture variations shown in FIGS. 7-10 for implementing aremote server computer comprising one or more compute servers and one ormore graphics servers are exemplary implementations. Nothing hereinshould be interpreted as limiting the disclosure to any particularimplementation aspect.

In a remote computing environment, a remote client computer typicallycommunicates with the compute server only and does not have a directconnection with the graphics server. The communication with the computeserver can comprise using a redirector and/or a broker similar to whatis shown in FIG. 3. The compute server typically manages the connectionwith the remote client computer. For example, the guest operating systemof the compute server can be configured to authenticate the client. Onceauthentication is complete, data transfer between the remote clientcomputer and the compute server can be initiated. As such, graphicsrequest from the remote client computer can be received by the computeserver. The compute server can process the graphics request intographics calls and commands and transmit these calls and commands to thegraphics server for rendering, capturing, and compression. Because thegraphics server does not have a direct communication path with theremote client computer, graphics output from the graphics server can besent to the compute server. The compute server can package and transmitthe graphics output to the remote client computer for display to anend-user.

Applying the above description in the virtual environment of FIG. 10,graphics requests from a remote client computer are transmitted over aphysical NIC 114 of a remote server computer 1000, though a virtual NIC(not shown) of a guest partition 1010, to an application 1012. Thegraphics request can then be processed and routed to a host partition1020 for rendering, capturing, and compressing. Once rendered, captured,and compressed, the output data can be routed back from the hostpartition 1020 to the compute server 1010 using the vGPU 1016. Totransmit the output data from the guest partition 1010 to the remoteclient computer, the guest partition 1010 can package and transmit thedata using the virtual NIC. The virtual NIC can redirect the output datato the physical NIC 114 that transmits the data over the network to theremote client computer. As described, the repetitive data traversalbetween the guest partition 1010, the host partition 1020, and theunderlying resources of the remote sever computer 1000 can requireintensive operations and can consume significant amount of memory andCPU resources that can increase the data delivery latency to the remoteclient computer.

FIG. 11 illustrates an alternative architecture that eliminates therepetitive data traversal between the resources of a remote servercomputer 1100. FIG. 11 describes an architecture that retains in acompute server 1110 the management of the connection with the remoteclient computer 1130 and that enables a graphics server 1120 to streamgraphics output data directly to the remote client computer 1130.Embodiments of the remote server computer 1100 may execute some or allof the components described with respect to remote server computer 700of FIGS. 7-9 and remote server computer 1000 of FIG. 10. Embodiments ofthe remote client computer 1130 may execute some or all of thecomponents described with respect to computer 100 of FIG. 1, remoteclient computer 210 of FIG. 2, and remote client computer 310 of FIG. 3.

The remote client computer 1130 can initiate a connection 1105 over anetwork (not shown) with the compute server 1110. The connection 1105can be TCPI/IP based and can comprise a remote desktop session, such asRDP 7.1. The compute server 1110 can authenticate the connection 1105with the remote client computer 1130. The authentication method canrequire the remote client computer 1130 to be authenticated before avirtual desktop or session for the remote client computer 1130 is set-upin the compute server 1110. The authentication method can be, forexample, a network level authentication available in RDP 7.1.Additionally, the compute server 1110 can acquire an address such as anIP address of the remote client computer 1130.

Once the authentication is complete and the connection 1105 between theremote client computer 1130 and the compute server 1110 is established,the compute server 1110 can initiate a remote desktop session, a virtualmachine, a desktop session within a virtual machine, or a combinationthereof running within the compute server 1110 for the remote clientcomputer 1130. Such virtual machine or desktop session can embody thetechniques of FIGS. 2-10. Additionally, the compute server 1110 caninitialize the graphics server 1120. The initialization can comprise,for example, restoring or waking-up the graphics server 1120 if thegraphics server is in an inactive state and establishing a connection1115 between the compute server 1110 and the graphics server 1120. Theconnection 1115 can be TCP/IP based or can use a VMBus in a virtualenvironment. The compute server 1110 can also provide the remote clientcomputer 1130 address to the graphics server 1120 and instruct thegraphics server 1120 to prepare for a connection request originatingfrom the remote client computer 1130 address. The compute server canalso acquire an address, such as an IP address, of the graphics server1120 and can provide the graphics server 1120 address to the remoteclient computer 1130.

Once the graphics server 1120 is initialized and the remote clientcomputer 1130 acquires the graphics server 1120 address, client computer1130 can initiate a connection 1125 over a network (not shown) with thegraphics server 1120. The connection 1125 can be TCP/IP based. Theconnection 1125 can also comprise a remote desktop session, such as RDP7.1. In a further embodiment, if the connection 1105 and the connection1125 comprise a remote desktop sessions, the remote desktop sessions canbe separate or can be the same across the connection 1105 and theconnection 1125.

Once the connections 1105, 1115, and 1125 are established, the remoteclient computer 1130 can send an end-user graphics input, such askeyboard or mouse input, to the compute server 1110. The computer servercan process the graphics input using the techniques of FIGS. 2, 6-10. Inone embodiment, the compute server 1110 can translate an end-userkeyboard strike or a mouse click on an icon into display commands thateffectuate an instance of an application and into display calls such asrotate, flip, resize, and draw. The compute server 1110 can send thedisplay commands and calls data to the graphics server 1120 forrendering, capturing, and compressing. The graphics server 1120 canrender, capture, and compress the display commands and calls data andencode the output as output graphics data. Instead of sending the outputgraphics data to the compute server 1110, the graphics sever 1120 cantransmit the output graphics data directly to the remote client computer1130 over the connection 1125. The remote client computer 1130 candecode the output graphics data for display to an end-user.

In an embodiment, where the remote server computer 1100 is virtualized,such as remote server computer 1000 of FIG. 10, the compute server 1110can be effectuated on a guest partition and the graphics server 1120 canbe effectuated on a host partition. Each partition can be configured tohave an address, such as an IP address. The address of the guestpartition or the computer server 1110 can be associated with a virtualNIC. The address of the host partition or the graphics server 1120 canbe associated with a virtual NIC or with a physical NIC 114 of theunderlying hardware of the remote server computer 1100. The computeserver 1110 can present a virtual machine or a session running thereinto the remote client computer 1130. The connection 1115 between thecompute server 1110 and the graphics server 1120 can comprise anintra-partition communication channel, such as a VMBus.

Graphics data such as an end-user input at the remote client computer1130 can be sent from the remote client computer 1130 over theconnection 1105 to the compute server 1110 address associated with thecomputer server 1110 virtual NIC.

The compute server 1110 can process the received graphics data intodisplay commands and calls data as previously described. The computeserver 1110 can send the processed graphics data to the graphics server1120 for rendering, capturing, and compressing. In an embodiment, thesending of the processed data can comprise transmitting the processeddata from the memory space allocated to the compute server 1110 to thememory space allocated to the graphics server 1120. In anotherembodiment, partitions within the remote server computer 1100 can sharememory space. In such embodiment, the sending of the data from thecompute server 1110 to the graphics server 1120 can comprise routing anaddress of a memory space containing the processed data rather thancopying the processed data between two memory spaces. For example, thecompute server 1110 and the graphics server 1120 can take advantage ofthe shared memory space within a VMBus to route the processed data.

The graphics server 1120 can render, capture, and compress the processeddata as previously described. The graphics server 1120 can encode therendered, captured, and compressed data into an output data and cantransmit the output data using the graphics server NIC to the addressassociated with the remote client computer 1130. The remote clientcomputer 1130 can decode the received output data for display to anend-user.

FIG. 12 depicts an embodiment of the architecture of FIG. 11 in anenvironment comprising a plurality of remote client computers 1130(A-N),a plurality of compute servers 1110(A-N), and a plurality of graphicsservers 1120(A-N). It is important to note that the numbers of remoteclient computers, compute servers, and graphic servers need not beequal. In other words, the relationships between remote client computer1130(A-N) and compute servers 1110(A-N), compute servers 1110(A-N) andgraphics servers 1120(A-N), and graphics servers 1120(A-N) and remoteclient computer 1130(A-N) can be one-to-one, one-to-many, or acombination thereof.

Embodiments of the remote server computer 1200 may execute some or allof the components described with respect to remote server computer 220of FIG. 2, remote server computer 320 of FIG. 3, remote server computer700 of FIGS. 7-10, remote server computer 1000 of FIG. 10, and remoteserver computer 1100 of FIG. 11. In one embodiment, the remote servercomputer 1200 can comprise a plurality of physical computing systems.Redirector and broker 1240 may execute some or all of the componentsdescribed with respect to redirector 330 and broker 340 of FIG. 3.Graphics server manager 1250 may execute some or all of the componentsdescribed with respect to graphics server manager 740 of FIG. 7, andgraphics server manager 850 of FIGS. 8-9. Additionally, the redirectorand broker 1240 can be integrated with the compute servers 1110(A-N).Similarly, the graphics server manager 1250 can be integrated with thegraphics servers 1120(A-N). In another embodiment, the remote servercomputer may be virtualized and may execute some or all of thecomponents described with respect to computer system 400 of FIGS. 4-5,remote server computer 1000 of FIG. 10, and remote server computer 1100of FIG. 11. In such embodiment, compute servers 1110(A-N) can beeffectuated on one or more guest partitions and graphics servers1120(A-B) can be effectuated on one or more host partitions. Theredirector and broker 1240 and the graphics server manager 1250 can alsobe virtualized in one or more guest partition and/or one or more hostpartition using the techniques previously described. In a furtherembodiment, the remote server computer 1200 can be a combination ofvirtual and physical machines. For example, the compute servers1110(A-N) and graphics servers 1120(A-N) may be virtualized while theredirector and broker 12040 and graphics server manager 1250 may bephysical computing devices.

The redirector and broker 1240 may be used to establish a firstconnection between a remote client computer 1130 and a compute server1110. For example, the redirector and broker 1240 may allocate a computeserver 1110 out of the plurality of compute servers 1110(A-N) and/or asession therein to the remote client computer 1130. As such, theredirector and broker 1240 can provide load balancing techniques withrespect to the availability of the compute server 1110(A-N). Once thatallocation is complete, the allocated compute server 1110 canauthenticate the remote client computer 1130. The remote client computer1130 and the allocated compute server 1110 can establish the firstconnection as previously described. For example, the first connectioncan comprise connection 1105 of FIG. 11. The allocated compute server1110 can also acquire an address of the remote client computer 1130.

The graphics server manager 1250 may be used to establish a secondconnection between the allocated compute server 1110 and a graphicsserver 1120 out of the plurality of graphics server 1120(A-N). Thegraphics server manager 1250 may execute load balancing techniques anddetermine the availability of the graphics servers 1120(A-N).Accordingly, the graphics server manager 1250 may then allocate agraphics server 1120 to the connection with the allocated computerserver 1110. Once the allocation is complete, the second connection canbe established. For example, the second connection can compriseconnection 1115 of FIG. 11.

The allocated compute server 1110 can provide the address of the remoteclient computer 1130 to the graphics server 1120 and can acquire andprovide the allocated graphics server 1120 address to the remote clientcomputer 1130. The remote client computer 1130 and the allocatedgraphics server 1120 can establish a third connection using the acquiredaddresses of the remote client computer 1130 and the allocated graphicsserver 1120. The third connection can comprise connection 1125 of FIG.11.

Further, load balancing techniques can be executed on the architectureof FIG. 12. For example, in case of load balancing of graphics servers1120(A-N), when the allocated graphics server 1120 can no longer servethe remote client computer 1130 properly, the compute server 1110 canestablish a new connection with an available graphics server 1120 andinstruct the remote client computer 1130 to establish a correspondingnew connection with the available graphics server 1120. The remoteclient computer 1130 can then seamlessly transition over to theavailable graphics server 1120. Additionally, in case of load balancingof compute servers 1110(A-N), when the allocated compute server 1110 canno longer serve the remote client computer 1130 properly, newconnections between the remote client computer 1130 and an availablecompute server 1110 and between the available compute server 1110 andthe already allocated graphics server 1120 can be established, while theconnection between the remote client computer 1130 and the alreadyallocated graphics server 1120 can remain the same. Techniques such asvirtual machine live migration can seamlessly transfer the workload fromone compute 1110 server to another while the graphics server 1120workloads can remain the same.

In an embodiment, techniques can be used to enable one or more remoteclient computers 1130 to accept display streams or graphics output datafrom various graphics servers 1120. For example, a remote clientcomputer 1130A can get permission from other remote client computers1130(B-N) to view the end-user displays associated with the other remoteclient computers 1130(B-N). The permission can comprise, for example,the addresses of the various graphics servers 1120 allocated to each ofthe other remote client computers 1130(B-N). Additionally, thepermission can comprise the addresses of the other remote clientcomputers 1130(B-N), the addresses of the compute servers 1110 allocatedto the each of the other remote client computers 1130(B-N), thecorresponding session IDs, user names, names of the compute servers 1110and graphics servers 1120, the number of active sessions in each computeserver 1110, and so on. The remote client computer 1130A can use thepermission to establish connections with the graphics servers 1120allocated to each of the other remote client computers 1130(B-N). Theconnections can comprise connection 1125 of FIG. 11. Once theconnections are established, the various graphics servers 1120 cantransmit the graphics data output of the other remote client computers1130(B-N) to the remote client computer 1130A. The remote clientcomputer 1130A can decode the graphics output data received from thevarious graphics servers 1120 and can display the decoded data to anend-user. For example, the remote client computer 1130A can display thedecoded data in a tiled fashion or can allow the end-user to flipthrough each of screens associated with the other remote clientcomputers 1130(B-N). Further, the remote client computer 1130A candisplay other information available in the permission, such as usernames, names of corresponding compute servers 1110 and graphics servers1120, and so on. In another embodiment, a graphics server 1120transmitting various graphics output data corresponding to remote clientcomputers 1130(B-N) can be configured to also transmit the variousgraphics output data and other information about the correspondingremote client computers 1130(B-N) to another remote client computer1130A. As such, the remote client computer 1130A can display the screensof and information about the remote client computers 1130(B-N).

FIG. 13 depicts an exemplary operational procedure for processinggraphics data for transmission to a client including operations 1300,1310, 1320, 1330, 1340, 1350, and 1360. Operation 1300 begins theoperational procedure and operation 1310 illustrates establishing afirst connection between a remote client and a compute server. The firstconnection can comprise connection 1105 of FIG. 11. Operation 1320illustrates establishing a second connection between the compute serverand a graphics server. The second connection can comprise connection1115 of FIG. 11. Operation 1330 illustrates establishing a thirdconnection between the remote client and the graphics server. The thirdconnection can comprise connection 1125 of FIG. 11. Operation 1340illustrates receiving graphics data by the compute server from theremote client over the first connection. The received graphics data cancomprise, for example, a user graphics input such as a keyboard strikeor a mouse click associated with the remote client. Operation 1350illustrates processing the received graphics data and sending theprocessed graphics data to the graphics server over the secondconnection. The processing of the received graphics data can comprisethe compute server translating the received graphics data into displaycommands and calls. Operation 1360 illustrates rendering, capturing,compressing, and transmitting the processed graphics data to the remoteclient over the third connection.

FIG. 14 depicts an exemplary system for processing graphics data fortransmission to a client computer as described above. System 1400comprises a processor 1410 and memory 1420. In an embodiment, theprocessor 1410 can be implemented as logical processing unit 102 in FIG.1, while the memory 1420 can be implemented as having some or all of thecomponents of the system memory 22 in FIG. 1. The memory 1420 furthercomprises computer instructions configured to cause the system toprocess graphics data for transmission to a remote client. Block 1422illustrates establishing a first connection between a remote client anda compute server. Block 1424 illustrates establishing a secondconnection between the compute server and a graphics server. Block 1426illustrates establishing a third connection between the remote clientand the graphics server. Block 1428 illustrates receiving graphics databy the compute server from the remote client over the first connection.Block 1430 illustrates processing the received graphics data and sendingthe processed graphics data to the graphics server over the secondconnection. Block 1432 illustrates rendering, capturing, compressing,and transmitting the processed graphics data to the remote client overthe third connection.

Any of the above mentioned aspects can be implemented in methods,systems, computer-readable media, or any type of manufacture.

The foregoing detailed description has set forth various embodiments ofthe systems and/or processes via examples and/or operational diagrams.Insofar as such block diagrams, and/or examples contain one or morefunctions and/or operations, it will be understood by those within theart that each function and/or operation within such block diagrams, orexamples can be implemented, individually and/or collectively, by a widerange of hardware, software, firmware, or virtually any combinationthereof.

It should be understood that the various techniques described herein maybe implemented in connection with hardware or software or, whereappropriate, with a combination of both. Thus, the methods and apparatusof the disclosure, or certain aspects or portions thereof, may take theform of program code (i.e., instructions) embodied in tangible media,such as floppy diskettes, CD-ROMs, hard drives, or any othermachine-readable storage medium wherein, when the program code is loadedinto and executed by a machine, such as a computer, the machine becomesan apparatus for practicing the disclosure. In the case of program codeexecution on programmable computers, the computing device generallyincludes a processor, a storage medium readable by the processor(including volatile and non-volatile memory and/or storage elements), atleast one input device, and at least one output device. One or moreprograms that may implement or utilize the processes described inconnection with the disclosure, e.g., through the use of an applicationprogramming interface (API), reusable controls, or the like. Suchprograms are preferably implemented in a high level procedural or objectoriented programming language to communicate with a computer system.However, the program(s) can be implemented in assembly or machinelanguage, if desired. In any case, the language may be a compiled orinterpreted language, and combined with hardware implementations.

While the invention has been particularly shown and described withreference to a preferred embodiment thereof, it will be understood bythose skilled in the art that various changes in form and detail may bemade without departing from the scope of the present invention as setforth in the following claims. Furthermore, although elements of theinvention may be described or claimed in the singular, the plural iscontemplated unless limitation to the singular is explicitly stated.

What is claimed:
 1. A system for processing data, comprising: at leastone computing device configured to at least: establish a connection withat least one client device and at least one graphics computing device;receive data from the at least one client device; process the receiveddata from the client device, and send the processed data to the at leastone graphics computing device, the processed data used to generategraphics data that is sent by the at least one graphics computing deviceto the client device.
 2. The system of claim 1, wherein the at least onecomputing device comprises at least one compute server and the at leastone graphics computing device comprises at least one graphics server. 3.The system of claim 1, wherein the at least one computing device isinstantiated within at least one virtual machine.
 4. The system of claim1, wherein the at least one graphics computing device is instantiatedwithin at least one partition on at least one virtualized machine. 5.The system of claim 3, wherein the at least one graphics computingdevice is instantiated within at least one partition on same physicalcomputing devices as the at least one computing device and wherein theconnection between the at least one computing device and the at leastone graphics computing device is at least one intra-partitioncommunication channel.
 6. The system of claim 1, wherein establishing aconnection with the at least one client device and the at least onegraphics computing device comprises acquiring at least one address ofthe at least one client device and at least one address of the graphicscomputing device, providing the acquired at least one client deviceaddress to the at least one graphics computing device and the acquiredat least one graphics computing device address to the at least oneclient device, and instructing the at least one client device and the atleast one graphics computing device to establish a connection using theprovided addresses.
 7. The system of claim 1, wherein establishing aconnection with the at least one client device and the at least onegraphics computing device comprises load balancing the at least onecomputing device and the at least one graphics computing device.
 8. Thesystem of claim 1, wherein the graphics data comprises a graphicsrepresentation of at least a portion of a remote desktop.
 9. The systemof claim 1, wherein the at least one computing device is executing atleast one remote desktop session with the at least one client device andwherein the graphics data comprises a graphic representation of at leasta portion of a user session.
 10. The system of claim 1, wherein theprocessed data comprises display commands and calls, and whereingenerating graphics data comprises rendering, capturing, and compressingthe processed data.
 11. The system of claim 1, wherein one or moregraphics computing device of the at least one graphics computing deviceare configured to send graphics data corresponding to multiple clientdevices to a single client device.
 12. A method for processing graphicsdata, comprising: establishing a connection between a plurality ofcomputing devices and a plurality of client devices and a connectionbetween the plurality of computing devices and a plurality of graphicsdevices; receiving data by the plurality of computing devices from theplurality of client devices; processing the received data; sending theprocessed data to the plurality of graphics devices; and instructing theplurality of graphics devices to generate graphics data using theprocessed data and to send the generated graphics data to the pluralityof client devices.
 13. The method of claim 12, wherein the plurality ofcomputing devices are instantiated within a plurality of virtualmachines, and the plurality of graphics devices are instantiated withina plurality of partitions on the same plurality of physical computingdevices as the plurality of computing devices.
 14. The method of claim13, wherein the plurality of computing devices are executing a pluralityof remote desktop sessions with the plurality of client devices and thegraphics data comprises a graphics representation of at least a portionof a user remote desktop session.
 15. The method of claim 12, whereinestablishing the connection between the plurality of computing devicesand the plurality of client devices and the connection between theplurality of computing devices and the plurality of graphics devicescomprises acquiring a plurality of client device addresses and providingthe acquired plurality of client device addresses to the plurality ofgraphics devices, acquiring a plurality of graphics device addresses andproviding the acquired plurality of graphics device addresses to theplurality of client devices, and instructing the plurality of clientdevices and the plurality of graphics devices to establish a connectionusing the provided addresses.
 16. The method of claim 12, wherein one ormore graphics devices of the plurality of graphics devices areconfigured to send graphics data corresponding to multiple clientdevices to a single client device.
 17. A computer readable storagemedium having stored thereon instructions that when executed by one ormore processors, cause the one or more processors to perform the stepsof: establishing a connection between a plurality of computing devicesand a plurality of remote client computers and a connection between theplurality of computing devices and a plurality of graphics computingdevices; receiving data by the plurality of computing devices from theplurality of client computers; processing the received data; sending theprocessed data to the plurality of graphics computing devices; andinstructing the plurality of graphics computing devices to generategraphics data using the processed data and to send the generatedgraphics data to the plurality of remote client computers.
 18. Thecomputer readable storage medium of claim 17, wherein the plurality ofcomputing devices are instantiated within a plurality of virtualmachines, and the plurality of graphics computing devices areinstantiated within a plurality of partitions on the same plurality ofphysical computing devices as the plurality of computing devices. 19.The method of claim 18, wherein the plurality of computing devices areexecuting a plurality of remote desktop sessions with the plurality ofremote client computers and the graphics data comprises a graphicsrepresentation of at least a portion of a user remote desktop session.20. The method of claim 17, wherein establishing the connection betweena plurality of computing devices and the plurality of remote clientcomputers and the connection between the plurality of computing devicesand the plurality of graphics computing devices comprises acquiring aplurality of remote client computer addresses and providing the acquiredplurality of remote client computer addresses to the plurality ofgraphics computing devices, acquiring a plurality of graphics computingdevice addresses and providing the acquired plurality of graphicscomputing device addresses to the plurality of remote client computers,and instructing the plurality of remote client computers and theplurality of graphics computing devices to establish a connection usingthe provided addresses.